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This invention relates to computer systems and, more 
particularly, to a network wherein multiple users may share, 
perceive, and manipulate a virtual environment generated by a 

10 computer system. 

Researchers have been working with virtual reality 
systems for some time. In a typical virtual reality system, 
people are immersed in three-dimensional, computer-generated 
worlds wherein they control the computer-generated world by 

15 using parts of their body, such as their hands, in a natural 
manner. Examples of virtual reality systems may be found in 
telerobotics, virtual control panels, architectural simulation, 
and scientific visualization. See, for example, Sutherland, 
W.R., "The Ultimate Display", Proceedings of the IPIP Congress 

2 0 2, 506-508 (1965); Fisher, S.S., McGreevy, M. , Humphries, J., 

and Robbinett, W. , "Virtual Environment Display System," Proc. 
86 Workshop 3D Graphics, 77-87 (1986) ; F.P. Brooks, 
"Walkthrough - A Dynamic Graphics System for Simulating Virtual 
Buildings", Proc. 1986 Workshop on Interactive 3D Graphics, 9- 
25 12 (1986); and Chung, J.C., "Exploring Virtual Worlds with 

Head-Mounted Displays", Proc. SPIE Vol. 1083, Los Angeles, CA, 
(1989) . All of the foregoing publications are incorporated 
herein by reference. 

In known systems, not necessarily in the prior art, a 

3 0 user wears a special helmet that contains two small television 

screens, one for each eye, so that the image appears to be 
three dimensional. This effectively immerses the user in a 
simulated scene. A sensor mounted on the helmet keeps track of 
the position and orientation of the users head. As the user's 
3 5 head turns, the computerized scene shifts accordingly. To 

interact with objects in the simulated world, the user wears an 
instrumented glove having sensors that detect how the hand is 
bending. A separate sensor, similar to the one on the helmet, 



r 
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determines the hand's position in space. A computer-drawn 
image of a hand appears in the computerized scene, allowing the 
user to guide the hand to objects in the simulation. The 
virtual hand emulates the movements of the real hand, so the 
5 virtual hand may be used to grasp and pick up virtual objects 
and manipulate them according to gestures of the real hand. An 
example of a system wherein the gestures of a part of the body 
of the physical user is used to create a cursor which emulates 
the part of the body for manipulating virtual objects is 
10 disclosed in copending U.S. Patent Application Ser v ial No. 



15 only a single user within the perceived virtual space. As a 

result, they cannot accommodate volitional virtual interaction 
between multiple users. 



reality network which allows multiple participants to share, 
perceive, and manipulate a common virtual or imaginary 
environment. In one embodiment of the present invention, a 
computer model of a virtual environment is continuously 

25 modified by input from various participants. The virtual 
environment is displayed to the participants using sensory 
displays such as head-mounted visual and auditory displays 
which travel with the wearer and track the position and 
orientation of the wearer's head in space. Participants can 

3 0 look at each other within the virtual environment and see 
virtual body images of the other participants in a manner 
similar to the way that people in a physical environment see 
each other. Each participant can also look at his or her own 
virtual body in exactly the same manner that a person in a 

3 5 physical environment can look at his or her own real body. The 
participants may work on a common task together and view the 
results of each other's actions. 




SUMMARY OF THE INVENTION 
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The present invention is directed to a virtual 
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BRIEF DESCRIPTION OF THE DRAWINGS 
^ is a diagram of a particular embodiment of a 
virtual realty network according to the present invention; 

Fig/\2 is a diagram of a data flow network for 
coupling refetj, world data to a virtual environment. 

Fig.^^ is a diagram showing three participants of a 
virtual reality experience; 

^ FigVs4 is a diagram showing a virtual environment as 
perceiveH^by one of the participants shown in Fig. 2; 
10 N FigN^5 is a diagram showing an alternative embodiment 

of a vrr^ual environment as perceived by one of the 
participantss^shown in Fig. 2; and 

Fig. 6^ is a flowchart showing the operation of a 
particular embodiment of a virtual reality network according to 
15 the present invention. 

BRIEF DESCRIPTION OF THE APPENDICES 
App. 1 is a computer program listing for the virtual 
environment creation module shown in Fig. 1; 
2 0 App. 2 is a computer program listing for the Data 

coupling module shown in Fig. 1; and 

App. 3 is a computer program listing for the visual 
display module shown in Fig. 1. 

2 5 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

Fig. 1 is a diagram showing a particular embodiment 
of a virtual reality network 10 according to the present 
invention. In this embodiment, a first participant 14 and a 
second participant 18 share and experience the virtual 
30 environment created by virtual reality network 10. First 
participant 14 wears a head-mounted display 22(A) which 
projects the virtual environment as a series of image frames 
much like a television set. Whether or not the helmet 
completely occludes the view of the real world depends on the 

3 5 desired effect. For example, the virtual environment could be 

superimposed upon a real-world image obtained by cameras 
located in close proximity to the eyes. Head-mounted display 
22(A) may comprise an EyePhone (TM) display available from VPL 
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Research, Inc. of Redwood City, CA. An electromagnetic source 
2 6 communicates electromagnetic signals to an electromagnetic 
sensor 30(A) disposed on the head (or head-mounted display) of 
first participant 14. Electromagnetic source 26 and 
5 electromagnetic sensor 30(A) track the position of first 

participant 14 relative to a reference point defined by the 
position of electromagnetic source 26. Electromagnetic source 
2 6 and electromagnetic sensor 30(A) may comprise a Polhemus 
Isotrak (TM) available from Polhemus Systems, Inc. Head- 
10 mounted display 22(A), electromagnetic source 26, and 

electromagnetic sensor 3 0(A) are coupled to a head-mounted 
hardware control unit 34 through a display bus 38(A), a source 
bus 42(A), and a sensor bus 46(A), respectively. 



15 assembly 50(A) which includes an electromagnetic sensor 54 for 
receiving signals from an electromagnetic source 58. 
Instrumented glove assembly 50(A), electromagnetic sensor 54(A) 
and electromagnetic source 58 are used to sense the position 
and orientation of instrumented glove 50 relative to a 

2 0 reference point defined by the location of electromagnetic 

source 58. In this embodiment, instrumented glove assembly 
50(A), electromagnetic sensor 54(A) and electromagnetic source 
58 are constructed in accordance with the teachings of co- 
pending patent application no. 317,107 entitled "Computer Data 
25 Entry And Manipulation Apparatus And Method." More 
particularly, instrumented glove assembly 50(A), 
electromagnetic sensor 54(A) and electromagnetic source 58 may 
comprise a DataGlove (TM) available from VPL Research, Inc. 
Instrumented glove assembly 50(A), electromagnetic sensor 

3 0 54(A), and electromagnetic source 58 are coupled to a body 

sensing control unit 62 through a glove bus 66, a sensor bus 
70, and a source bus 74, respectively. 



in Fig. 1, it should be understood that the position and 
3 5 orientation of any and all parts of the body of the user may be 
sensed. Thus, instrumented glove 50 may be replaced by a full 
body sensing suit such as the DataSuit (TM) , also available 
from VPL Research, Inc., or any other body sensing device. 



First participant 14 also wears an instrumented glove 



Although only an instrumented glove assembly is shown 
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In the same manner, second participant 18 wears a 
head-mounted display 22(b) and an electromagnetic sensor 30(b) 
which are coupled to head-mounted hardware control unit 3 4 
through a display bus 38(b) and a sensor bus 46(b), 
5 respectively. Second participant 18 also wears an instrumented 
glove assembly 50(b) and an electromagnetic sensor 54(b) which 
are coupled to body sensing control unit 62 through a glove bus 
66(b) and a sensor bus 70(b). 

In this embodiment, there is only one head-mounted 

10 hardware control unit 34, body sensing control unit 62, 

electromagnetic source 26, and electromagnetic source 58 for 
both participants. However, each participant may be located 
separately from each other, in which case each participant 
would have his or her own head-mounted hardware control unit 

15 34, body sensing control unit 62, electromagnetic source 26, 
and/or electromagnetic sensor 58. 

The position and orientation information received by 
head-mounted control unit 34 are communicated to a virtual 
environment data processor 74 over a head-mounted data bus 76. 

2 0 Similarly, the position and orientation information received by 
body sensing control unit 62 are communicated to virtual 
environment data processor 74 over a body sensing data bus 80. 
Virtual environment data processor 74 creates the virtual 
environment and superimposes or integrates the data from head- 

2 5 mounted hardware control unit 34 and body sensing control unit 

62 onto that environment. 

Virtual environment data processor 74 includes a 
processor 82 and a virtual environment creation module 84 for 
creating the virtual environment including the virtual 

3 0 participants and/or objects to be displayed to first 

participant 14 and/or second participant 18. Virtual 
environment creation module 84 creates a virtual environment 
file 88 which contains the data necessary to model the 
environment. In this embodiment, virtual environment creation 
3 5 module 84 is a software module such as RB2 SWIVEL (TM) , 

available from VPL Research, Inc. and included in app. 1. 

A data coupling module 92 receives the virtual 
environment data and causes the virtual environment to 




dynamically change in accordance with the data received from 
head-mounted hardware control unit 34 and body sensing control 
unit 62. That is, the virtual participants and/or objects are 
represented as cursors within a database which emulate the 
position, orientation, and other actions of the real 
participants and/or objects. The data from the various sensors 
preferably are referenced to a common point in the virtual 
environment (although that need not be the case) . In this 
embodiment, data coupling module 92 is a software module such 
as BODY ELECTRIC (TM^, available from VPL Research, Inc. and 
included in app. 2. ^^9- 2 shows an example of a simple data 
flow network for coupling data from the head of a person in the 
real world to their virtual head. Complex interactions such as 
hit testing, grabbing, and kinematics are implemented in a 
similar way. The data flow network shown in Fig. 2 may be 
displayed on a computer screen and any parameter edited while 
the virtual world is being simulated. Changes made are 
immediately incorporated into the dynamics of the virtual 
world. Thus, the participants are given immediate feedback 
about the world interactions he or she is developing 

The animated virtual environment is displayed to 
first participant 14 and second participant 18 using a virtual 
environment display processor 88. In this embodiment, virtual 
environment display processor 88 comprises one or more left eye 
display processors 92, one or more right eye display processors 
96, and a virtual display module 100. In this embodiment, each 
head-mounted display 22(a), 22(b) has two display screens, one 
for each eye. Each left eye display processor 92 therefore 
controls the left eye display for a selected head mounted 
display, and each right eye display processor 96 controls the 
right eye display for a selected head mounted display. Thus, 
each head mounted display has two processors associated with 
it. The image (viewpoint) presented to each eye is slightly 
different so as to closely approximate the virtual environment 
as it would be seen by real eyes. Thus, the head mounted 
displays 22(A) and 22(B) produce stereophonic images. Each set 
of processors 92, 96 may comprise one or more IRIS (TM) 
processors available from Silicon Graphics, Inc. 
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The animated visual environment is displayed by a 
series of image frames presented to each display screen within 
head-mounted displays 22(a) and 22(b). These frames are 
computed by a visual display module 100 which runs on each 
5 processor 92, 96. In this embodiment, visual display module 
108 comprises a software module such as ISAAC (TM) , available 
from VPL Research, Inc. and included in app. 3. 

In this embodiment, only the changed values within 
each image frame are communicated from processor 82 to left eye 

10 display processors 92 and right eye display processors 96 over 
an Ethernet bus 108. After the frames for each eye are 
computed, a synchronization signal is supplied to processor 82 
over a hard-sync bus 104. This informs processor 82 that the 
next image frame is to be calculated, and processor 82 then 

15 communicates the changed values needed to calculate the next 

frame. Meanwhile, the completed image frames are communicated 
to head-mounted hardware control unit 34 over a video bus 112 
so that the image data may be communicated to head-mounted 
displays 22(a) and 22(b). 

2 0 Fig. 3 is a diagram of virtual reality network 10 as 

used by three participants 12 0, 124 and 128, and Figs. 3 and 4 
provide examples of the virtual environment as presented to two 
of the participants. As shown in Figs. 3-5, participants 12 0 
and 124 engage in a common activity whereas participant 128 

2 5 merely watches or supervises the activity. In this example, 

and as shown in Figs. 4 and 5, the activity engaged in is an 
engineering task on a virtual machine 132 wherein virtual 
machine 132 is manipulated in accordance with the corresponding 
gestures of participants 120 and 124. Fig. 4 shows the virtual 

3 0 environment as displayed to participant 120. Of course, the 

other participants will see the virtual environment from their 
own viewpoints or optical axes. In this embodiment, the 
actions of the participants shown in Fig. 3 are converted into 
corresponding actions of animated participants 120(A), 124(A) 
35 and 128(a), and the virtual environment is created to closely 
match the real environment. 

A unique aspect of the present invention is that the 
appearance and reactions of the virtual environment and virtual 
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participants are entirely within the control of the user. As 
shown in Fig. 5, the virtual environment and actions of the 
virtual participants need not correspond exactly to the real 
environment and actions of the real participants. Furthermore, 
5 the virtual participants need not be shown as humanoid 

structures. One or more of the virtual participants may be 
depicted as a machine, article of manufacture, animal, or some 
other entity of interest. In the same manner, virtual machine 
13 2 may be specified as any structure of interest and need not 

10 be a structure that is ordinarily perceivable by a human being. 
For example, structure 132 could be replaced with giant 
molecules which behave according to the laws of physics so that 
the participants may gain information on how the molecular 
world operates in practice. 

15 It should also be noted that the real participants 

need not be human beings. By using suitable hardware in 
processor 82, such as the MacADIOS (TM) card available from GW 
Instruments, Inc. of Summerville, Mass., any real-world data 
may be modeled within the virtual environment. For example, 

2 0 the input data for the virtual environment may consist of 

temperature and pressure values which may be used to control 
virtual meters displayed within the virtual environment. 
Signals from a tachometer may be used to control the speed of a 
virtual assembly line which is being viewed by the 

2 5 participants. 



desired. For example, participant 128 could share the 
viewpoint of participant 120 (and hence view his or her own 
actions) , and the viewpoint could be taken from any node or 
30 perspective (e.g., from virtual participant 120(A) 's knee, from 
atop virtual machine 132, or from any point within the virtual 
environment) . 



particular embodiment of virtual reality network 10. The 
3 5 virtual environment is created in a step 2 00, and then nodes on 
the virtual objects within the virtual environment are defined 
in a step 204. The raw data from head-mounted hardware control 
\ unit 34 and body sensing control unit 62 are converted to 



Viewpoints (or optical axes) may be altered as 



Fig. 6 is a flowchart illustrating the operation of a 




position and orientation values in a step 208, and the position 
and orientation data is associated with (or coupled to) the 
nodes defined in step 204 in a step 212, Once this is done, 
processors 92 and 96(a) may display the virtual objects (or 
participants) in the positions indicated by the data. To do 
this, the viewpoint for each participant is computed in a step 
216, The system then waits for a synchronization signal in a 
step 218 to ensure that all data necessary to compute the image 
frames are available. Once the synchronization signal is 
received, the image frame for each eye for each participant is 
calculated in a step 220. After the image frames are 
calculated, they are displayed to each participant in a step 
224. It is then ascertained in a step 228 whether any of the 
nodes defined within the virtual environment has undergone a 
position change since the last image frame was calculated. If 
not, then the same image frame is displayed in step 224. If 
there has been a position change by at least one node in the 
virtual environment, then the changed position values are 
obtained from processor 82 in a step 232. It is then 
ascertained in a step 234 whether the virtual environment has 
been modified (e.g., by changing the data network shown in Fig. 
2) . If so, then the virtual object nodes are redefined in a 
step 236. The system again waits for a synchronization signal 
in step 218 to prevent data overrun (since the position and 
orientation values usually are constantly changing) , and to 
ensure that the views presented to each eye represent the same 
information. The new image frames for each eye are then 
calculated in a step 220, and the updated image frames are 
displayed to the participants in a step 224. ^ 3 

While the above is a complete description of a 
preferred embodiment of the present invention, various 
modifications and uses may be employed. For example, the 
entire person need not be simulated in the virtual-environment. 
For the example shown in Fig. 1, the virtual environment may 
depict only the head and hands of the virtual participant. 
Users can communicate at a distance using the shared 
environment as a means of communications. Any number of users 
may participate. Communications may take the form of speech or 





10 



other auditory feedback including sound effects and music; 
gestural communication including various codified or impromptu 
sign languages; formal graphic communications, including 
charts, graphs and their three-dimensional equivalents; or 
5 manipulation of the virtual environment itself. For example, a 
window location in the virtual reality could be moved to 
communicate an architectural idea. Alternatively, a virtual 
tool could be used to alter a virtual object, such as a virtual 
chisel being used to chip away at a stone block or create a 
10 virtual sculpture. 



resources for creation and improvement of the virtual reality. 
Data may be shared, such as a shared anatomical data base 
accessible by medical professionals and students at various 
15 locations. Researchers at different centers then could 

contribute then different anatomical data to the model, and 
various sites could contribute physical resources to the model 
(e.g., audio resources, etc.). 



2 0 virtual reality network to practice theatrical or other 

performing arts. The virtual reality network may provide 
interactive group virtual game environments to support team and 
competitive games as well as role playing games. A virtual 
classroom may be established so that remotely located students 

25 could experience a network training environment. 



time animation, or to eliminate the effects of disabilities by 
the participants. Participants with varying abilities may 
interact, work, play and create using individualized input and 
3 0 sensory display devices which give them equal abilities in the 
virtual environment . 



presented to the user using first and second audio displays to 
produce the experience that the source of the sound is located 
35 in a specific location in the environment (e.g., from the mouth 
of a virtual participant) , and three-dimensional images may be 
presented to the participants. 



A virtual reality network allows the pooling of 



Participants in the expressive arts may use the 



The virtual reality network also may be used for real 



Stereophonic, three-dimensional sounds may be 
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Linking technology for remotely located participants 
include Ethernet, phoneline, broadband (ISDN) , and satellite 
broadcast, among others. Data compression algorithms may be 
used for achieving communications over low bandwidth media. If 
5 broadband systems are used, a central processor may process all 
image data and send the actual image frames to each 
participant. Prerecorded or simulated behavior may be 
superimposed on the model together with the real time behavior. 
The input data also may come from stored data bases or be 

10 alogorithimically derived. For example, a virtual environment 
could be created with various laws of physics such as 
gravitational and inert ial forces so that virtual objects move 
faster or slower or deform in response to a stimulus. Such a 
virtual environment could be used to teach a participant how to 

15 juggle, for example. 

Other user input devices may include eye tracking 
input devices, camera-based or others input devices for sensing 
the position and orientation of the real world participants 
without using clothing-based sensors, force feedback devices as 

20 disclosed in U.S. Patent Application No. 315,252 entitled 

"Tactile Feedback Mechanism For A Data Processing System" filed 
on February 21, 1989 and incorporated herein by reference, 
ultrasonic tracking devices, infrared tracking devices, 
magnetic tracking devices, voice recognition devices, video 

2 5 tracking devices, keyboards and other conventional data entry 

devices, pneumatic (sip and puff) input devices, facial 
expression sensors (conductive ink, strain gauges, fiber optic 
sensors, etc.), and specific telemetry related to the specific 
environment being simulated, i.e., temperature, heart rate, 

3 0 blood pressure, radiation, etc. Consequently, the scope of the 

invention should not be limited except as described in the 
claims. 
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